Conclusion

PSCI 2270 - Week 13

Georgiy Syunyaev

Department of Political Science, Vanderbilt University

November 21, 2024

Plan for this week



  1. Presentations

  2. c

  3. Q&A and feedback

Presentations

Reminders


  • Final presentations are during class after Thanksgiving

    • Each of you will have 8 minutes for presentation with 3 minutes of feedback
    • Announcement on Brightspace with sign up link today
    • There will be pizza and drinks!
  • Final PAPs are due on OSF (and link to it on Brightspace) by December 10

Final presentations


  • 5-7 minutes \(\approx\) 5-7 slides

  • Do not put too much text on the slides:

    • \(6 \times 6\) rule: Unless absloutely unavoidable don’t put more than 6 words and 6 lines per slide
    • Try not to read from slides
  • What to include: Motivation, Research Question, Hypotheses, Research Design (Context/Unit of analysis/Experiment or Observational), Independent/Dependent variables, Measurement and procedures, Possible issues

PAPs

Pre-analysis plans

  • Why do you think we need them?

  • PAPs help avoid

    • \(p\)-hacking: We do not know what the authors planned originally and if they selectively reported only significant results
    • Publication bias: Journals select publications based on results rather than quality
    • Low quality research: Force you to think ahead of time about possible issues and address them
    • Low impact research: Given the costs of making PAP, it forces you to think hard about your treatment

\(p\)-hacking and publication bias (Brodeur, Cook, and Heyes 2020)

What should go into PAP

1. Clear hypothesis(es)

  • explanation of (general) theory
  • stating testable hypotheses in terms of your study

2. Primary dependent variables and how to measure them

  • what raw measures are used and how they are transformed into outcomes

3. Primary independent/treatment variables and how to measure them

  • what raw measures are used and how they are transformed into independent variables
  • this includes confounders that you plan to control for

4. Precise statistical model to be tested

  • what comparisons will you look at
  • how you will calculate main estimates (e.g. difference in means, regression, etc.)
  • how will you calculate uncertainty

PAPs in practice (Ofosu and Posner 2023)

Resources



  • Let’s look at example on Brightspace

Class summary

Foundations of political science research

What we covered

  • Core steps of research project: Theory, measurement, causal inference
  • Possible issues at each step: non-positive claims, measurement errors, biases in causal identification
  • Practice: Examples of how social scientists form hypotheses, address measurement and causal inference issues

Why is it useful

  1. Enhances the ability to critically evaluate any research results
  2. Brings you one step closer to being able to implement your own research projects independently
  3. Provides reference material for when you will need it

Key lessons



  • Causal inference is crucial (even in the age of big data and ChatGPT)
  • Think about any relationships not just in terms of estimate but also in terms of uncertainty
  • Be careful with interpreting associations as causal relationship
  • In your own research be transparent about why and what you do and know weaknesses of your research design

Questions? Feedback?

References

Brodeur, Abel, Nikolai Cook, and Anthony Heyes. 2020. “Methods Matter: P-Hacking and Publication Bias in Causal Analysis in Economics.” American Economic Review 110 (11): 3634–60.
Ofosu, George K, and Daniel N Posner. 2023. “Pre-Analysis Plans: An Early Stocktaking.” Perspectives on Politics 21 (1): 174–90.